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1.0  Summary 


This  effort  enhanced  Robotics  STEM  activities  by  incorporating  cognitive  tutors  at  key  points  to  make 
important  mathematicai  decision  or  impiement  criticai  caicuiations.  The  program  utiiized  Cognitive 
Tutor  Authoring  toois  for  designing  probiem  soiving  tasks,  estabiishing  prototypicai  soiutions,  reasoning 
about  soiution  steps,  and  cognitive  modeis.  The  project  deveioped  new  technoiogies  designed  to 
increase  students  computationai  and  aigorithmic  thinking.  The  project  targeted  both  formai  and 
informai  education.  The  deveioped  Computer  Science  Student  Network  (CS2N)  teaches  muitipie  aspects 
of  computer  science  in  scaffoid  ways  that  students  and  teachers  wiii  find  engaging. 

2.0  introduction 

The  Computer  Science  Student  Network  (CS2N)  was  funded  by  DARPA  using  Air  Force  Research 
Laboratory  Award  FA8750-10-2-0165.  The  project's  goai  is  to  significantiy  increase  the  number  of 
students  studying  Computer  Science  -  Science,  Technoiogy,  Engineering  and  Mathematics  (CS-STEM). 
The  project  is  the  resuit  of  CMU's  Fostering  innovation  through  Robotics  Expioration  (FiRE)  project.  The 
project  is  housed  are  Carnegie  Meiion  University's  Robotics  Academy  and  uses  the  Robotics  Academy's 
infiuence  in  the  Robotics  Education  community  to  market  CS-STEM  activities  nationaiiy. 


3.0  Project  Goais  -  Fostering  innovation  through  Robotics  Expiorations  (FiRE) 

Figure  1  provides  the  stated  goai  for  the  Fostering  innovation  through  Robotics  (FiRE)  project; 
“Inspire  IM  students  per  year  to  pursue  advanced  CS-STEM  careers.”  Based  on  formai  and 
informai  education  adoption  of  FiRE  technoiogies  the  project  can  ciaim  that  it  is  reaching  neariy 
1  miiiion  chiidren  per  year.  FiRE  inspired  curricuium  and  software  is  being  used  in  over  10,000 
formai  and  informai  education  environments.  Robot  Virtuai  Worid  software  has  been 
downioaded  over  100,000  times,  has  been  purchased  by  over  1,000  schoois,  and  is  growing  in 
popuiarity  with  the  Robotics  Competition  Community  which  has  over  35,000  US  teams. 

FiRE  technoiogies  have  been  studied  in  ciassroom  environments  and  have  proven  to 
significantiy  increase  mathematicai  abiiities  in  chiidren  (Expedition  Atiantis  Study)  and  improve 
the  rate  chiidren  iearn  to  program  robots  by  60%  (Robot  Virtuai  Worid  Study).  There  is  stiii 
much  work  to  do  to  improve  adoption  as  weii  as  efficacy,  but  project  technoiogies  have  been 
successfuiiy  impiemented  and  proven  to  be  effective  in  formai  and  informai  education  settings. 
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Inspiring  1,000,000  students  per  year  to  pursue  advanced  CS-STEM  careers 


Fostering  Innovation  through  Robotics  Exploration 

The  Project  Starts  With  +  DARPA  Funding  ~  The  Nation  Gets 


Innovative  Educational  Technology 

Cognitive  Tutor  Authoring  Tools 
1  5  years  Development,  S30M  Investment 

Robotics  Academy  Curriculum 
10  years  Development,  S10M  Investment 
Robotics  Academy  Networking  and  Outreach 

Alice  Programming  Language 
12  years  Development,  $10M  Investment 
Alice  Proiect  Networking  and  Outreach 

World  Class  Researchers  and  Faculty 

CMU  School  of  Computer  Science 

The  Robotics  Institute 

The  Human  Computer  Interaction  Institute 

University  of  Pittsburgh's 
J  Learning  Research  and  Development  Center 

World  Class  Distribution  Networks 

Informal  Educotion  Distribution  Networks 

Robotics  Competitions  -  FIRST,  VEX,  BotBall, 

BEST,  RoboFest,  the  Technology  Student 
Association  -  consist  of  over  20,000  teams 

4H,  Boys  and  Girl  Scouts, 

Boys  and  Girls  Clubs,  YWCAA^MCA  -  many 
organizations  use  Carnegie  Mellon  training  tools 

An  Established  Commercial  Distribution  Network 

LEGO  Education,  Innovation  First  Corp.,  Robomatter  Inc. 

Carnegie  Mellon 


A  Self  Sustaining 

CS-STEM  National 
Resource  Center 

Mission  statement: 

"To  inspire  and  prepare  generations  of 
American  students  to  pursue  CS-STEM  careers." 

that  Inspires  and  Promotes  CS-STEM  Excellence  for  Students 

-»•  Cognitive  Tutor  Enabled  Cvberlearninq  Training  Tools 

that  teach  mathematical  competency,  computer 
programming,  and  algorithmic  thinking  through  robotics. 

^Web-Enabled  Tools  that  Track  Student's  CS-STEM 

activities,  draws  them  into  other  CS-STEM  activities,  and 
provides  personal  and  social  incentives  as  they  progress 
from  project  to  project. 

-»-An  Autonomous  Multi-Robot  Competition 
Designed  to  teach  multi-agent  communications,  shared  decision 
making,  resource  allocation,  and  disciplined  engineering  process. 

Robot  Virtual  World  "Game-Like"  Simulation  Tool 

that  enables  students  to  write  robot  programs  using  robot 
competition  software  and  test  their  robots  in  a  virtual  world. 

-»-An  Alice  Animation  Competition 

that  fits  into  existing  organizational  structures,  including  robotic 
competitions,  where  students  create  3D  stories,  animations, 
marketing  materials  and  games.  Alice  has  been  proven  very 
effective  at  reaching  girls  and  maintaining  CS  interest. 

-*•  New  computational  thinking  activities  and  curriculum 


Figure  1  -  Project  Goal 


4.0  FIRE  became  the  Computer  Science  Student  Network 

The  Computer  Science  Student  Network  (CS2N),  depicted  in  Figure  2,  is  an  online  aggregator  of 
robotics  and  computer  science  related  activities.  CS2N's  project  design  starts  with  activities  in 
informal  education  like:  robotics  competitions.  Boy  Scout  merit  badges,  and  4H  robotics. 
Carnegie  Mellon  has  iteratively  improved  these  robotics  technologies,  aligned  them  with 
standards,  developed  teachers'  guides,  and  made  them  available  to  formal  and  informal 
education. 
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Carnegie  Mellon  is  creating  a 


COMPUTER  SCIENCE  SOCIAL  NETWORK 


using  strategies  found  in  online  games. 


STARTING 

WITH  ROBOTICS  COMPETITIONS 


\/=x 


BESf 


INCLUDING 

INFORMAL  EDUCATION  NETWORKS 


CHANGING  _ 

FORMAL  EDUCATION 

Y  A  A  HOB 


Figure  2.  Computer  Science  Student  Network 


5.0  Lessons  Learned  -  FIRE's  Proudest  Accomplishments 

The  project  developed  a  robotic  simulation  environment  that  enables  children  and  schools 
without  physical  robots  to  learn  programming.  The  simulation  environments  are 
programmable  using  LEGO  NXT-G,  LEGO  EV3,  LabVIEW,  and  ROBOTC  software  and  LEGO 
NXT,  TETRIX,  EV3  and  VEX  Cortex  and  VEXIQ  robots;  these  are  the  dominant  software  and 
hardware  environments  found  in  classrooms  today.  Carnegie  Mellon  Robotics  Academy 
(CMRA)  will  continue  to  study  and  iteratively  improve  on  these  "game  changing" 
technologies  since  they  have  been  found  to  be  an  effective  way  to  teach  computational 
thinking  and  are  orders  of  magnitude  less  expensive  that  their  physical  counterparts. 

The  Expedition  Atlantis  math  game  -  The  Expedition  Atlantis  (EA)  game  was  developed  to 
replace  the  Robotics  Flash-based  Cognitive  Tutor  when  the  project  team  determined  that 
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the  limitations  of  a  Flash-based  development  environment  would  limit  the  scalability  of  our 
new  robot-math  focused  training  materials.  EA  is  developed  using  Unity,  a  game 
developments  software,  that  is  compatible  with  iPads,  Android  devices,  PCs  and  MACs.  EA 
testing  in  classrooms  found  that  when  properly  implemented  robotics  can  be  used  to  teach 
mathematics.  EA  in  now  available  on  iPads  in  the  form  of  an  app,  the  game  is  fun  to  kids, 
and  EA  is  part  of  CS2N's  long  term  sustainability  plan. 

The  development  of  a  cloud-based  aggregator,  CS2N  that  enables  project  staff  to:  offer 
online  training,  collect  data,  enables  teachers  to  setup  classes,  offers  automatic  assessment 
in  the  form  of  badges,  and  enables  all  stakeholders  to  track  students'  progress  through 
either  CS2N  or  the  student's  local  computer.  CS2N  will  enable  CMRA  to  continue  to  conduct 
research  on  new  educational  technologies  that  it  develops. 

The  development  of  teacher  and  student  certifications.  Over  the  life  of  the  FIRE  project 
CMRA  project  staff  developed  a  data  bank  of  assessment  questions  that  aligned  with  the 
competencies  taught  in  a  college  entry-level  computer  science  course.  Those  questions, 
along  with  additional  robotics  questions  enabled  the  project  to  develop  a  Robotic 
Educator's  Certification  (REC)  for  the  LEGO  and  VEX  hardware  platforms.  The  certification 
test  measures  content  knowledge  as  well  as  a  teacher's  pedagogical  abilities.  At  the  end  of 
teacher  training  the  teacher  takes  the  REC  test.  If  the  teacher  passes  the  REC  test,  they 
receive  a  REC  that  certifies  that  they  have  the  skills  to  teach  robotics  at  the  middle  or  high 
school  level. 

The  ability  to  offer  free  training  through  multi-media  developed.  Highly  scaffold  curriculum 
which  was  developed  with  FIRE  (and  other)  funding.  The  ongoing  development  of  free 
Moodle  Based  courses  that  are  used  by  students  every  day.  CS2N  supports  thousands  of 
students  and  teachers  competing  in  robotics  competitions  each  robotic  competition  season. 

CMRA  was  able  to  leverage  DARPA  funding  to  secure  DML  funding,  NSF  funding,  and 
corporate. 

CMRA  has  a  sustainability  plan  that  will  leverage  DARPA  funding  to  sustain  CMRA  outreach 
for  the  foreseeable  future  (which  was  a  significant  DARPA  requirement). 

6.0  Lessons  Learned  FIRE's  Biggest  Disappointments 

The  project  staff  initially  decided  to  develop  training  materials  around  a  Flash-based 
Cognitive  Tutor  (CT)  that  is  no  longer  being  used  because  Flash  is  not  supported  by  iPads. 
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The  good  news  is  the  demise  of  Flash  pushed  us  toward  game  development  software  and 
that  the  lessons  learned  during  the  development  of  the  CT  transferred  into  the  new  game. 

That  the  project  has  developed  incredible  educational  technologies  like: 

a.  A  cloud-based  multi-robot  online  robotics  competition  that  will  enable  kids  from 
schools  across  the  country  to  compete  virtually; 

b.  A  robot  transformer  that  could  enable  students  to  import  their  own  custom  robots 
into  the  robot  simulation  environment; 

c.  Automated  assessment  tools  designed  to  help  teachers  to  understand  what  children 
know  or  don't  know; 

d.  And,  a  cloud-based  computer  science/math  game  that  shows  significant 
mathematical  learning  gains  in  testing,  and  leads  into  CS  programming,  that  needs 
extension  activities  as  well  as  further  testing  to  improve  its  efficacy  in  formal  and 
informal  settings. 

Research  shows  that  these  technologies  show  incredible  promise,  but  CMRA  doesn't  have  the 
funding  to  complete  the  research  and  development  around  them. 

7.0  References  -  Published  Research  and  Dissemination 

FIRE  partnerships  with  organizations  like  the  Boy  Scouts,  4H,  the  Robotics  Educational 
Foundation,  FIRST  Robotics,  Robomatter,  and  BEST  Robotics  have  enabled  the  CMRA/DARPA 
partnership  to  make  a  significant  difference  in  the  lives  of  hundreds  of  thousands  of  children. 
The  three  biggest  enables  of  CS-STEM  literacy  are:  the  curriculum  which  is  available  on  line  at 
no  charge,  the  Robot  Virtual  World  software  (RVW)  which  CMRA  makes  available  at  no  charge 
each  summer  for  training  purposes,  and  the  Expedition  Atlantis  math/computer  science  game 
which  shows  other  developers  how  to  foreground  an  academic  concept  in  a  robotics  context. 
Directly  below  is  the  FIRE  Projects  published  research  and  presentations: 
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Pennsylvania  Annual  Conference,  Camp  Hill  Pennsylvania.,  [  PDF  copy  of  PowerPoint  ] 

3.  Plot,  J.,  Shoop,  R  (November  2013)  Teaching  Programming  with  Robot  Virtual  Worlds.. 
Presentation  given  at  the  Technology  Education  and  Engineering  Association  of  Pennsylvania 
Annual  Conference,  Camp  Hill  Pennsylvania.,  [  PDF  copy  of  PowerPoint  ] 
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Systems  to  Boost  Achievement  Motivation.  2011  GLS  Conference  [Paper  PDF] 

Approaches  in  Teaching  Mathematics  and  Robotics 

1.  King,  S.,  Stein,  M.,  Schunn,  C.D.,  (2012,  May). Designing  Educative  Guides:  Reconceptualizing 
Teacher's  Role  in  Teacherless  Cognitive  Tutor-based  Robotics  Instruction.Paper  presented  at  the 
2012  annual  meeting  of  the  American  Society  for  Engineering  Education,  Vancouver,  BC.  [Paper 
(PDF)] 

2.  Silk,  E.  M.  (2011). Resources  for  learning  robots:  Environments  and  framings  connecting  math  in 
robotics(Doctora\  dissertation.  University  of  Pittsburgh).  Available  from  D-Scholarship  at  the 
University  of  Pittsburgh.  (No.  8607)  [Paper  (PDF)]  [Presentation  (PDF)] 

3.  Silk,  E.  M.,  Higashi,  R.,  &  Schunn,  C.  D.  (2011,  iune). Resources  for  robot  competition  success: 
Assessing  math  use  in  grade-school-level  engineering  design.  Paper  to  be  presented  at 
theannual  meetingof  theAmerican  Society  for  Engineering  Education,  Vancouver,  BC,  Canada. 
[Paper  (PDF)]  [Presentation  (PDF)] 

4.  Silk,  E.  M.,  &  Schunn,  C.  D.  (2011,  iur\e).Calculational  versus  mechanistic  mathematics  in 
propelling  the  development  of  physical  knowledge.  Paper  to  be  presented  at  the41st  annual 
meetingof  theJean  Piaget  Society,  Berkeley,  CA,  USA.  [Paper  (PDF)]  [Presentation  (PDF)] 

5.  Silk,  E.  M.,  &  Schunn,  C.  D.  (2011,  AprW). Resources  for  learning  robots:  Facilitating  the 
incorporation  of  mathematical  models  in  students'  engineering  design  strategies.Paper  to  be 
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presented  at  theannual  meetingof  theAmerican  Educational  Research  Association,  New  Orleans, 
LA,  USA.  [Paper  (PDF)]  [Presentation  (PDF)] 

6.  Silk,  E.  M.,  Schunn,  C.  D.,  Shoop,  R.,  &  Stein,  M.  K.  (2011,  March). f/ie  Robot  Algebra  Project. 
Poster  presented  at  theeighth  annual  NSF  ITEST  Summit,  Arlington,  VA,  USA.  [Poster  (PDF)] 

7.  Silk,  E.  M.  (2010,  August  25).  Can  math  help  in  LEGO  robotics  competitions?  [4-partweb  logpost). 
Retrieved  fromhttp://robotics-academv.org/blog/?p=356[Part  1]  [Part  2]  [Part  3]  [Part  4] 

8.  Silk,  E.  M.,  Higashi,  R.,  Shoop,  R.,  &  Schunn,  C.  D.  (2010).  Designing  technology  activities  that 
teach  mathematics.  The  Technology  Teacher,  69  (4),  21-27.  [Paper  (PDF)] 

8.0  CMRA  FIRE/CS2N  Sustainability  Plan 

CMRA  will  use  the  following  strategies  to  sustain  the  DARPA  seed  funded  mission: 

Teacher  Training -The  Robotics  Academy  offers  summer  institutes  each  summer  and  online 
training  each  semester.  The  training  generates  approximately  $150K  after  expenses  per  year. 

Teacher  Certification  Tests  -  In  2013,  $70K  was  generated  through  certification  courses.  These 
are  fees  that  teachers  pay  in  order  to  take  the  certification  courses. 

Student  Certification  Courses  -  We  will  develop  mini-courses  that  teach  "Computational 
Thinking  Practices"  and  applied  Science,  Technology,  Engineering,  and  Mathematics".  The  mini¬ 
courses  will  be  4-6  weeks  in  length  and  teach  students  how  to  program  using  games  and  Robot 
Virtual  World  (RVW)  technologies.  Each  course  will  be  part  of  a  mapped  badged  pathway  that 
lead  to  either  an  industry  recognized  certification  (i.e.  Certified  LabVIEW  Associate  Developer) 
or  to  College  in  the  High  School  Credit.  The  certification  mini  courses  will  be  offered  via  the 
cloud  with  a  live  instructor  and  an  asynchronous  learning  infrastructure  that  is  supported  using 
CS2N  developed  technology  and  infrastructure.  We  will  offer  the  course  through  existing 
networks  (i.e.  FIRST  LEGO  League,  cyber-schools,  and  virtual  charter  schools).  The  mini-courses 
will  be  available  for  purchase  at  $30  per  seat  and  will  include  all  the  software  a  student  will  need 
to  take  the  course  as  well  as  access  to  an  instructor  lead  course.  We  are  testing  the  curriculum, 
software,  and  CS2N  infrastructure  for  the  course  this  summer  during  our  Robotics  Summer  of 
Learning  Project. 

Robot  Virtual  World  Software  has  been  licensed  to  Robomatter  Inc.  a  for-profit  company  that  is 
an  offshoot  of  Carnegie  Mellon's  Robotics  Academy.  Robomatter  was  spun  out  of  Carnegie 
Mellon's  Robotics  Academy  in  2004  and  has  made  its  technology  and  IP  available  to  the  Robotics 
Academy  at  no  charge.  Robomatter  will  generate  approximately  $250K  of  revenue  from  sales  of 
RVW  during  2014  which  will  pay  for  full  time  developers  and  support  personnel  to  support  the 
software.  Robomatter  projects  this  revenue  stream  will  increase  significantly  as  more  teachers 
and  students  become  aware  of  the  software. 

Expedition  Atlantis  App  -  Currently,  Expedition  Atlantis  is  available  for  free  for  teachers  to  use 
with  PCs  and  Macs.  Robomatter  paid  for  the  development  of  an  iPad  App  that  will  be  marketed 
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through  the  Apple  Store.  Since  we've  never  produced  or  sold  Apps  in  the  past  we  have  no 
experience  that  allows  us  to  predict  sales  figures  for  the  near  future.  Having  said  that  when  we 
searched  for  Robotics  Apps  the  Apps  that  were  there  were  not  very  good  and  we  expect  to 
generate  revenue  from  the  sale  of  this  App.  We  will  market  the  App  to  schools  and  sell  it  at  bulk 
pricing,  (i.e.  100  annual  licenses  for  $250).  We  believe  that  all  App  sales  to  education  will  need 
to  be  annual  licenses  because  we  will  need  to  update  the  App  as  the  devices  update  and  evolve. 

Ongoing  Educational  Research  -  DARPA  funding  has  enabled  the  Robotics  Academy  to  develop 
technologies  that  are  ripe  for  emerging  cloud-based  educational  training.  The  Robotics  Academy 
will  continue  to  seek  research  opportunities  that  enable  it  to  refine  and  test  these  technologies. 

Partnerships -The  Robotics  Academy  will  continue  to  seek  partnerships  with  like-minded 
organizations  looking  to  improve  computer  science  and  STEM  education.  One  partnership 
involved  Carnegie  Innovations,  a  Carnegie  Mellon  owned  for-profit  company  that  licenses 
Carnegie  Mellon  IP  globally. 

9.0  Conclusions  -  FIRE  Project  Summation 

Carnegie  Mellon  Robotics  Academy's  FIRE  project  has  created  incredible  technology  as  well  as 
produced  significant  research  that  will  propel  the  development  of  CS-STEM  based  activities  in 
the  formal  and  informal  education  community.  These  technologies  are  finding  their  way  into 
informal  and  formal  education  across  the  United  States.  Tools  built  during  the  FIRE  project  will 
continue  as  part  of  the  CMRA  outreach.  Additionally,  CMRA  will  continue  to  use  the  technology 
developed  by  the  FIRE  project  to  offer  training,  certifications,  and  to  collect  data  for  research. 
Revenue  generated  by  this  technology  will  help  sustain  Robotics  Academy  outreach  and 
research  and  development. 
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10.0  Acronyms 

CMRA  -  Carnegie  Mellon  Robotics  Academy 

CMU  -  Carnegie  Mellon  University 

CSSN  -  Computer  Science  Student  Network 

CS-STEM  -Computer  Science  -  Science,  Technology,  Engineering  and  Mathematics 
CT  -  Cognitive  Tutor 

DARPA-  Defense  Advanced  Research  Projects  Agency 
EA  -  Expedition  Atlantis 

FIRE  -  Fostering  Innovation  through  Robotics  Exploration 
NSF  -  National  Science  Foundation 
REC  -  Robotic  Educator's  Certification 
RVW  -  Robot  Virtual  World  software 
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